RTM at SemEval-2016 Task 1: Predicting Semantic Similarity with Referential Translation Machines and Related Statistics

We use referential translation machines (RTMs) for predicting the semantic similarity of text in both STS Core and Cross-lingual STS. RTMs pioneer a language independent approach to all similarity tasks and remove the need to access any task or domain specific information or resource. RTMs become 14th out of 26 submissions in Cross-lingual STS. We also present rankings of various prediction tasks using the performance of RTM in terms of MRAER, a normalized relative absolute error metric. 1 Semantic Agreement We participated in Semantic Textual Similarity task at SemEval-2016 (Bethard et al., 2016) with RTMs. RTMs identify translation acts between any two data sets with respect to interpretants, data close to the task instances, effectively judging monolingual and bilingual similarity. We use RTMs for predicting the semantic similarity of text. Interpretants are used to derive features measuring the closeness of the test sentences to the training data, the difficulty of translating them, and the presence of the acts of translation, which may ubiquitously be observed in communication. Semantic Web’s dream is to allow machines to share, exploit, and understand knowledge on the web (Berners-Lee et al., 2001). As more and more shared conceptualizations of domains emerge, we get closer to this goal. Semantic textual similarity (STS) task (Agirre et al., 2016) at SemEval2016 (Bethard et al., 2016) is about quantifying the degree of similarity between two given sentences S1 and S2 in the same language (English) in STS Core (STS English) or in different languages (English or Spanish) in Cross-lingual STS (STS Spanish), with a real number in [0, 5]. S1 and S2 may be constructed using different models and with different conceptualizations of the world or different ontologies and different vocabulary. Even if two instances are categorized as same, they may have different implications for commonsense reasoning (both albatros and penguin are a bird) (Biçici, 2002). The existence of a single ontology that can cover all the required conceptual information for reaching semantic understanding is questionable because it would presume an agreement among all ontology experts. Yet, semantic agreement using heterogeneous ontologies may not be possible as well since in the most extreme case, they would not use the same tokens. Therefore, semantic textual similarity is harder than the Chinese room thought experiment (Internet Encyclopedia of Philosophy, 2016) since we are not given any instructions about how to answer queries. Our goal is to quantify the level of semantic agreement between S1 and S2 and RTMs use interpretants, data close to the task instances for building prediction models for semantic similarity. 2 Referential Translation Machine Each RTM model is a data translation prediction model between the instances in the training set and the test set and translation acts are indicators of the data transformation and translation. RTMs are powerful enough to be applicable in different domains and tasks while achieving top performance in both Figure 1: RTM depiction: ParFDA selects interpretants close to the training and test data using parallel corpus in bilingual settings and monolingual corpus in the target language or just the monolingual target corpus in monolingual settings; an MTPPS use interpretants and training data to generate training features and another use interpretants and test data to generate test features in the same feature space; learning and prediction takes place taking these features as input. ans.-ans. headlines plagiarism postediting que.-que. STS base 1572 1498 1271 3287 1555 English eval. 254 249 230 244 209 multisource newswire STS base 2973 301 Spanish eval. 294 301 Table 1: Number of instances in the STS test set. Only some of the instances are actually evaluated (eval. row). monolingual (Biçici and Way, 2015) and bilingual settings (Biçici et al., 2015b). Our encouraging results in the semantic similarity tasks increase our understanding of the acts of translation we ubiquitously use when communicating and how they can be used to predict semantic similarity. Figure 1 depicts RTMs and explains the model building process. Given a training set train, a test set test, and some corpus C, preferably in the same domain, the RTM steps are: 1. select(train,test, C)→ I 2. MTPP(I,train)→ Ftrain 3. MTPP(I,test)→ Ftest 4. learn(M,Ftrain)→M 5. predict(M,Ftest)→ ŷ RTMs use ParFDA (Biçici et al., 2015a) for instance selection and machine translation performance prediction system (MTPPS) (Biçici and Way, 2015) for generating features. We use support vector regression (SVR) for building the predictor in combination with feature selection (FS) and partial least squares (PLS). Assuming that ŷ, y ∈ Rn are the prediction and the target respectively, evaluation metrics we use are defined in Equation (1) where metrics are Pearson’s correlation (r), mean absolute error (MAE), relative absolute error (RAE), relative Pearson’s correlation (rR), MAER (mean absolute error relative), and MRAER (mean relative absolute error relative). We use MAER and MRAER for easier replication and comparability. MAER is the mean absolute error relative to the magnitude of the target and MRAER is the mean absolute error relative to the absolute error of a predictor always predicting the target mean assuming that target mean is known (Biçici and Way, 2015). b . c caps its argument from below to where = MAE(ŷ, y)/2, which represents half of the score step with which a decision about a change in measurement’s value can be made. Domain r Model ans-ans. headlines plagiarism postediting que.-que. Weighted r r rR MAE RAE MAER MRAER SVR 0.4486 0.6634 0.8038 0.8133 0.6237 0.6685 0.6506 0.7563 1.015 0.679 0.5819 0.726 PLS-SVR 0.344 0.6605 0.8064 0.8231 0.6454 0.6518 0.6386 0.7786 1.0228 0.684 0.5779 0.739 FS+PLS-SVR 0.3533 0.6529 0.8049 0.823 0.648 0.6524 0.6369 0.7733 1.0243 0.685 0.5766 0.742 Table 2: STS English test results for each domain. Domain r Model Multisource r News r Weighted r Rank r rR MAE RAE MAER MRAER FS+PLS-SVR 0.5204 0.5915 0.5564 14 0.5244 0.5291 1.241 0.809 0.8812 0.856 SVR 0.5294 0.4985 0.5137 16 0.4455 0.4075 1.3473 0.878 0.9933 0.924 FS-SVR 0.5284 0.536 0.5322 15 0.4691 0.444 1.3094 0.853 0.9441 0.891 Table 3: STS Spanish test results.